Remote control with a single control button

ABSTRACT

An apparatus and method for the remote control and/or interaction-with electronic-devices such as computers; home-entertainment-systems; media-centers; televisions; DVD-players; VCR-players; music systems; appliances; security systems; toys/games; and/or displays. A user may orient a pointer (e.g., laser pointer) to place a visible pointer-spot on/near object(s) on an active-display(s); and/or a fixed-display(s); and/or on real-world object(s) within a display region or pointer-spot detection-region. Detectors and/or camera(s) may be connected/attached to the display region and/or a structure that is connected/attached to display region. When the user initiates a “select”, the detectors/cameras may detect the location of the pointer-spot within the display region. Corresponding to the user&#39;s selection(s); control action(s) may be performed on the device(s) being controlled/interacted-with and additional selection-menus may be optionally presented on an active-display. In an alternative embodiment, the invention may operate equivalent to a computer-mouse or other pointing-device. In another alternative embodiment, the invention may be made inter-operable with existing electronic-devices [e.g., televisions] that are controlled with infrared signaling.

BACKGROUND OF INVENTION

1. Field of the Invention

This invention relates to the remote control-of and/or interaction-withelectronic-devices such as computers; home-entertainment-systems;media-centers; televisions; DVD-players; VCR-players; music systems;displays; appliances; security systems; games/toys; or any other type ofdevice that user(s) may control or interact with.

2. Description of Related Art

Current remote controls are complex to use because of the large numberof control-buttons they have. Some remotes have over 50 physicalbuttons. In addition, some remotes utilize multi-button functionalcombinations which further increase the user complexity. Many users donot know what all the buttons do or how to use them. At any given time,only a portion of the buttons may be functional; but the presence ofthese non-functional buttons makes it more difficult for the user tofind the button for the desired function.

The large quantity of buttons leads to small size buttons that aretightly packed (e.g., large fingers) and small size labels (e.g.,farsighted) that present additional difficulties for certain users. Thesmall size buttons often result in text-abbreviations or small-iconsthat the user has difficulty seeing and/or deciphering their intendedfunction. In addition, the user may be using the remote-control in adarkened room, where limited ambient light may make it difficult to findthe desired buttons.

In addition, in many applications, the user is required to continuallyswitch between nearsighted tasks (of looking at the control buttons) andfarsighted tasks (of looking up at a display) which may present range offocus/correction problems for certain users.

In addition, if the user presses the wrong button, the remote-controlmay enter a mode that the user doesn't know how to get out of or toaccidentally make changes to the set-up configuration that the user maynot know how to undo or must expend great effort to determine how toundo.

In addition, the user must use specific navigation buttons such as up;down; right; and/or left to move the selection-point on the screen. Theuser often needs to press the navigation buttons multiple times and in aparticular order to arrive at the desired selection point. In addition,the user may need to alternately look up to the screen and back-down tothe navigation and/or other control buttons multiple times; to checkthat the navigation movements or control actions are happening asexpected and the intended action is occurring.

Some prior art has used an illumination-pointer (e.g., laser-pointer).For example, a camera may be mounted separately and remotely from thedisplay, in-order to “see” the pointer's spot. A major disadvantage of aseparately mounted camera is that the difficulty in setting-up,adjusting, aligning and calibrating the camera to be able to see andrecognize a specific region. The need to set-up a separate cameraincreases the user complexity and increases the time and effort requiredbefore the user can begin using their purchase(s).

In addition, if the camera's view is incorrectly set-up or is laterdisturbed, the camera may not be able to “see” the region of interestwith sufficient accuracy.

In addition, there are many situations where the desired location(s) fora remotely located camera are undesirable from the user's perspectivedue to aesthetic or other competing uses for that location (e.g., in ahome family-room or living-room).

In addition, in some configurations and applications, other users oreveryday objects may temporarily or suddenly block the remotely locatedcamera's view. For example, a user may stand or place a real-worldobject (e.g., a chair) between the remotely located camera and thedisplay. When this happens, the system will suddenly stop working andthe user may not realize why and become frustrated. The user may blamethe system for being unreliable and wish to return their unit.

In addition, if the user wants to move the display to a new location orto different viewing angle, the remote camera must also be moved andthen the set-up process repeated again. This significantly increases theuser complexity and prevents the user from quickly moving their displayto other locations (e.g., another room) or to other viewing angles(e.g., turning or tilting the display to prevent glare or so othersusers can also see the display).

In addition, those skilled in the art will realize that there are manyother disadvantages of using a separately positioned camera to view thepointer's spot on a display.

What is desired is a simpler way of remotely controlling and/orinteracting-with electronic-devices.

SUMMARY OF INVENTION

Detectors and/or camera(s) may be attached and/or embedded into adisplay or display structure so that a pointer-spot (e.g.,laser-pointer) may be detected when it is at any location within adefined detection-region. The detection-region may includeactive-display(s) and/or fixed-display(s) and/or real-world objects thatthe user may select or interact-with.

A user may orient a pointer to place the pointer-spot on/nearselectable-object(s) present on an active-display or a fixed-display ora real-world object, that are within the detection-region 921. When theuser initiates a “select”, the detectors and/or imager(s) [e.g.,cameras] may sense the location of the pointer-spot within thedetection-region 921 [e.g., display region]. The captured-image(s)[e.g., the detector responses] may be analyzed to determine thepointer-spot location(s) and to determine which object(s) the userselected [e.g., the object(s) the user was pointing at]. Thedesired-action(s) that correspond to the user's selection(s) may beperformed on the device(s) being controlled and/or other actions takenthat correspond with the user's intention.

The detectors and imager(s) may be aligned and calibrated with thedisplay region 921 so the location of the pointer-spot within thedisplay region may be determined. In one embodiment of the invention,one or more cameras may be mounted-to and/or integrated-with the displayregion or a structure that attaches to the display region. In anotheralternative embodiment, the detectors are distributed across theactive-display(s) and/or fixed-displays and/or real-world-objects in thedisplay region.

The invention may be used as a remote-control for one or moreelectronic-devices (e.g., televisions; DVD-players; music-systems;appliances). The invention may also be used as a computer-mouse orpointing-device, in-order to interact with objects on an active-display(such as a computer).

In some embodiments, a dedicated control-display may be used. In otherembodiments, the control-display may be shared with other user-devicefunctions (e.g., a television-display may be temporarily used as acontrol-display) and/or use only a portion of the active-display 101.

A single control-display may be used to control and/or interact-with aplurality of electronic-devices. Alternatively, a plurality ofcontrol-displays may be located in multiple locations [e.g., around ahome or business] where each control-display may be used to control allor any subset of the devices. The invention may also be used to controldevices remotely across the Internet or other Wide-Area-Network (e.g.,home devices may also be controlled from work or other locations).

A plurality of user-devices that are to be controlled, may beadded/removed by automatically detecting their attachment/removal on anetwork or communication-backbone via plug-and-play and/or standarddevice-discovery techniques.

In an alternative embodiment, the detectors and/or imager(s) may bemounted [and calibrated] onto a frame-assembly. The frame-assembly maythen be attached to active and/or fixed display(s); either during themanufacturing process or at a later time [e.g., retro-fitted to anexisting display in a user's home].

Some objectives and advantages of the invention [depending on theembodiment] include:

-   -   (1) Intuitively simple for a user to use.    -   (2) Eliminate the need for navigation buttons such as up; down;        left; right.    -   (3) Reduce or eliminate the need for the user to look-down at        the controller to find the correct button to push and then back        up to the display to see if it happened.    -   (4) Reduce the number of controls/buttons the user must deal        with. In some embodiments, only one control and/or button is        needed.    -   (5) Simplify user choices/selections to only those that are        valid possibilities in the current context.    -   (6) Work with any type of active-display and/or fixed-display        and/or everyday real-world-object.    -   (7) Reduce the amount of user set-up and configuration.

Some additional objectives and features of the invention [depending onthe embodiment] include:

-   -   (1) A single remote-controller may be used to control multiple        devices.    -   (2) Automatically adapt as devices are added/removed from a        user's system.    -   (3) May be implemented as a “universal” remote-controller.    -   (4) May be implemented to be backwardly compatible with older        electronic-devices that are controllable via infrared        control-codes.

BRIEF DESCRIPTION OF DRAWINGS

The invention will be better understood by reference to the detaildescription taken in conjunction with the accompanying drawings.

FIG. 1 illustrates an overview of the remote-control functionality forone embodiment.

FIG. 2 a illustrates one embodiment of a remote-controller with acylinder-type physical configuration.

FIG. 2 b illustrates one embodiment of a remote-controller with arectangular-type physical configuration.

FIG. 2 c illustrates one embodiment of a remote-controller with aring-type physical configuration that can be worn on a user's finger.

FIG. 3 shows the functional steps for one embodiment of using apointer-spot to control an electronic-device.

FIG. 4 a illustrates an example of selectable-objects on anactive-display that the user may select using the remote-controller'spointer.

FIG. 4 b illustrates examples of various fixed-objects (e.g., physicalcontrols or fixed-images) that the user may select using a pointer-spot.

FIG. 7 a shows a simplified example of a table that may be used topre-define what actions are to be taken, when selectable-objects areselected in each display-image or display-region.

FIG. 7 b shows a simplified table that may be used to define thelocation of each selectable-object in a display-region or control-menu.

FIG. 8 a shows a region 875 of the active-display where the reflectedpointer-spot is simultaneously visible to one specific detector incamera 131 and one specific detector in camera 132.

FIG. 8 b shows a cross-sectional view 821 (see FIG. 8 a) of thedisplay-frame and the active-display surface showing an example wherethe cameras are located above the active display surface [e.g., displayregion].

FIG. 9 a shows an example utilizing a line of detectors, along the X andY axis, that are attached to an active-display or display-frame ordisplay region.

FIG. 9 b shows an example utilizing a single area-view camera that isattached to an active-display or display-frame or display region.

FIG. 9 c shows an example utilizing multiple cameras positioned alongboth the X and Y axis of the active-display or display-frame or displayregion.

FIG. 9 d shows an example utilizing cameras attached near the fourcorners of the active-display or display-frame or display region.

FIG. 9 e shows an example of distributing detectors [that are capable ofdetecting the pointer-spot] across the area of an active-display.

FIG. 9 f is a detailed view of region 931 illustrating a distribution ofdetectors across the display area and the relative detector andpointer-spot sizes.

FIG. 10 shows an example where there are three active-displays 101 and afixed-display 109 containing a plurality of fixed-objects, locatedwithin a display region [e.g., pointer-spot detection region] defined bya display-frame.

FIG. 11 shows one embodiment of a flow diagram for the automaticdiscovery and set-up [plug-and-play]; when the user adds a newelectronic-device to the system.

FIG. 12 shows one embodiment of a processing flow chart for determiningand performing a user's intended action.

FIG. 13 shows a block diagram of an embodiment of an RF-to-InfraredConverter.

DETAIL DESCRIPTION

The present invention is now described with reference to the drawings.

Functional Overview Simplified Example

FIG. 1 illustrates a functional overview of one particular embodiment ofremote-control and/or interactive-control. The remote-controller 105 mayinclude a pointer 102 that emits light-beam 111 that may form apointer-spot 112 on/near objects within a pointer-spot detection-region921. The user may orient the remote-controller 105 to move thepointer-spot 112 in-order to select and/or interact-with objects thatappear within the pointer-spot detection-region 921.

In the embodiment shown in FIG. 1, an active-display 101 is within thepointer-spot detection-region 921. In the embodiment shown in FIG. 1,detectors and/or imagers (cameras 131 and 132) may be aligned to thedisplay so that the imagers may detect the pointer-spot 112 when it isat any location with the display 101. In the particular embodiment shownin FIG. 1, the cameras (131 & 132) have an approximately 90 degreeangular field-of-view (135 and 136) that allows the cameras to see thepointer-spot reflecting/scattering off any location withinactive-display 101.

Based upon operational context, the processing 106 may placecontrol-menus and/or selection-options on the active-display 101, thatthe user may select and/or interact with. The control-menus and/orselection-options may be used to control one or more devices that arebeing controlled 108 [e.g., electronic-devices and/or user-devices]. Inthe example shown in FIG. 1, four objects [“option 1” 121; “option 2”122; “option 3” 123; and “option 4” 124] may be [temporarily] displayedon the active-display 101 and represent options that the user mayoptionally select from.

To select an option, the user may activate the pointer-beam 111 and thenorient the remote-controller 105 to place the pointer-spot 112 on [ornear] one of the objects and then initiate a “select”. The user mayinitiate a “select” via the user-control-interface 104. In oneembodiment, the user-control-interface 104 may include control-button(s)203 such as one or more mechanical switches that the user may press. Forexample, in one particular embodiment, a single switch/button may act asa very simple user-control-interface 104. As shown in FIG. 1, theuser-control-interface 104 may be included in the remote-controller 105.The activation of a “selection”, may be communicated 114 to theprocessing 106.

When the “select” is activated, the processing 106 may command thedetectors and/or imager(s) to acquire one or more images that will allowthe location of the pointer-spot to be determined. The captured-imagesmay then be communicated 115 to the processing 106 where the image(s)are analyzed to determine the location of the pointer-spot at the timethe “select” was activated. Using the pointer-spot location, theprocessing may then determine which selectable-object [e.g.,menu-option] the user “selected”. The processing 106 may determine thecontrol-commands that need to be issued the appropriate device(s) 108in-order to perform the control-action(s) that correspond to the user's“selection”. The control-commands may be communicated 115 to thedevice(s)-being-controlled 108. Depending on the application andcontext, the processing may then cause a new set of menu-options orselectable-object(s) or entertainment programming to be displayed to theuser on the active-display 101.

In some embodiments, media drive(s) 117 (e.g., a DVD or CD) and/orInternet/Network 118 interface may be optionally attached to theprocessing 106; in-order to provide [one-time] set-up and/orconfiguration-data and/or other information useful for controlling andinteracting with various electronic-devices 108.

More Generally:

Some configurations may have multiple active-displays 101 and/ordetection-regions 921 that a user may place the pointer-spot 112 within.For example, the user may control all the devices in their home throughany of the active-displays 101 distributed in several different rooms oftheir home. The user may own multiple remote-controllers 105 that may beinterchangeably used with any of the active-displays and/ordetection-regions 921.

In general, the user may select and/or interact with objects that arewithin the pointer-spot detection-region 921. In general, detectorsand/or imager(s) may be attached/embedded in a display and/ordisplay-frame so that they may see the pointer-spot 112 at any locationwithin a pointer-spot detection-region 921. In general, the pointer-spot112 may be used to designate and/or interact with object(s) onactive-display(s) 101; fixed-display(s) 109; and/or real-world-objects.

In one embodiment, one or more imagers (e.g., cameras) may beintegrated-with and/or attached to an active-display 101 and/or adisplay-frame 941; in order to detect the pointer-spot 112 onactive-display(s) 101 and/or on fixed-object(s) 109 [e.g.,real-world-object(s)].

In an alternative embodiment, detectors may be distributed [e.g., FIGS.9 e and 9 f] across the region of an active-display(s) 101 and/or acrossfixed-object(s) 109 [e.g., real-world-object(s)]; in order to detect thepointer-spot 112.

By attaching and aligning the cameras and/or detectors to anactive-display 101 and/or a display-frame during manufacturing, the useris not burdened with mounting; set-up; alignment and calibrationassociated with separately mounted camera(s). In some retro-fitembodiments, the user may acquire a display-frame with camera(s) and/orembedded detectors. The display frame 941 may then be attached to analready owned active-display 101 or used to create a detection-region921.

In general, the user may use the remote-controller 105 to control anytype of electronic device. A single controller may also be used tocontrol a plurality of devices 108 and/or interact with multipleactive-displays 101 and/or fixed-displays 109. The remote-controller maybe used to drag; pull and/or interact-with objects across multipleactive-displays 101 and take actions on one screen that affect anotherscreen.

In many embodiments, no surface or working area is needed to operate theremote-controller 105; even when performing the functions equivalent tothose of a computer-mouse. This allows the user to easily orient theremote-controller from any position, including standing; sitting;reclining; lying on a couch; etc.

The invention may be used over a wide range of distances from theremote-control 105 to the active-display(s) 101 and/or fixed-object(s).In one embodiment, the invention may be used to controlling devices thatare across a room in a work; home or other setting.

The invention may also be used to interact with fixed oractive/interactive real-world objects such as a billboard ordisplay-advertisements.

In some embodiments, the invention may allow the simultaneous use ofmultiple remote-controllers by many different and/or unrelated users.

Remote-Controller Embodiments:

The remote-controller 105 may include a pointer 102; auser-control-interface 104; and circuitry to allow communication 114with the processing 106.

The remote-controller 105 may be integrated and sized to be easilyoriented by the user to point the aperture-side of the remote-controllertoward objects-of-interest. In some embodiments, the remote-controllermay be mobile and portable. In some embodiments, the remote-controllermay be configured to be easily held and oriented by the user. In someembodiments, the remote-controller may be transported in a user's pocketor worn by the user.

In one embodiment, as shown in FIG. 2 a, the remote-controller 105 mayhave an approximately cylindrical-type shape, similar to a laser pointeror a pen. The pointer's output aperture 201 may be located and alignedat one end of the remote-controller. The remote-controller 105 mayoptionally contain a user-control-interface 104, such ascontrol-button(s) 203. In one embodiment, the control-button(s) 203 maybe mechanical-type switch(s) that the user may easily press with thethumb of the hand that is holding the remote-controller. Theremote-controller's cross-section may be approximately circular orelliptical and may be variable along its length. The remote-controllerscross-section may be configured so that the control-button(s) arenaturally aligned with the users thumb when held in the user's hand.

In one embodiment, the remote-controller 105 may have a singlecontrol-button 203 with three positions: “off”; “beam-on”; and “select”switch positions. The control-button 203 may be configured toautomatically return to the “off” position when the user stopspressing-on the control-button 203 or sets down the remote-controller105.

The shape of the remote-controller 105 may also be configured to preventthe remote-controller 105 from rolling when it is set down onto a flator smooth surface (e.g., a table top). For example, theremote-controller may optionally contain a clip 205; that prevents theremote-controller from rolling on a flat surface and also allows theremote-controller to be clipped onto a shirt pocket or other objects.

In another embodiment shown in FIG. 2 b, the remote-controller 105 mayhave a square or rectangular cross-section [e.g., similar in shape toremote-controllers currently used for televisions; DVD-players;music-systems; etc]. The pointer's output aperture 201 may be locatedand aligned at one end of the remote-controller 105. Theremote-controller 105 may also contain a user-control-interface 104,such as the control-button(s) 203.

In one embodiment, the control-button(s) 203 may be easily and naturallyreached by the thumb of the same hand that is holding theremote-controller 105.

In some embodiments, the functions of remote-controller 105 may be addedinto many other existing devices. For example, the remote-controller 105functions [102, 104, 114] may be incorporated into a pen; pencil;standard laser-pointer; cell-phone; personal digital assistant (PDA); orany other (mobile) device.

In another alternative embodiment, the remote-controller 105 functions[102, 104, 114] may be incorporated into any type of prior-art remotecontrollers. For example, the remote-controller 105 functions [102, 104,114] may be also incorporated into a many-button remote-controller. Insome embodiments, this may ease the transition from prior-art remotecontrollers to this invention since a user may continue to operate theold way or use the new invention, as desired.

In some other embodiments, the elements of the invention may be sharedwith other elements/capabilities that are normally included in otheruser-devices. For example, a pointer 102 may be added to cell-phones orpersonal digital assistant (PDA) or other mobile device. The processing106 [or a portion of the processing] may also be shared in thesemulti-function devices. A pointer such as a laser-pointer or otherlight-emitting pointer may be incorporated into the phone. Abutton-switch or any other type of user-control may be used to turn-onthe pointer and/or to “select” objects. The software may be imbedded inthe phone and/or on other processors located elsewhere across acommunication path 114 such as the cell-phone network and/or theInternet connection.

In another embodiment, the remote-controller may be configured to beworn by the user. In one embodiment as shown in FIG. 2 c, theremote-controller 105 may be shaped like a ring may be worn on a finger(e.g., the index finger). The remote-controller 105 may be ring-shapedwith a hole 231 for sliding onto a user's finger. The size of the ringopening may be made to be adjustable to any size finger by using anelastic-band and/or adjustable Velcro-type strips for part of the ring.The pointer's output aperture 201 may be located at one end of the ringso it points away from the wearer's wrist. The remote-controller 105 mayalso contain a user-control-interface 104, such as control-button(s)203. If the remote-controller shown in FIG. 2 c is worn on theright-hand index finger then the right-hand's thumb (of the same handthat is wearing the ring) may easily activate the control-button(s) 203.The ring may also contain ornamentation as desired.

Many other wearable configurations are also possible. For example, theremote-controller function may be:

-   -   Embedded in a wrist-like device such as a bracelet or a watch        worn by the user    -   Embedded in the user's clothing.    -   Embedded or attached to the user including implanted in the        skin.

In an optional alternative embodiment, the remote controller may bemounted on a stand or tripod so it may be easily accessible to the userand may be easily oriented to point at objects-of-interest.

User-Control-Interface Implementations:

In general, the user may communicate with the processing 106 via theuser control-interface 104. Depending on the embodiment, the usercontrol-interface 104 may utilize physical-controls such as buttons;switches; scroll-wheels; keyboards; etc. In other embodiments,verbal/audio-commands and/or any other method of inputting auser-command (e.g., “select”) may be used. In some embodiments, theuser-control-interface may implement the functional equivalent of acomputer-mouse [e.g., with equivalents to the right and left mousebuttons; scroll-wheel; etc.].

In some embodiments, as shown in FIG. 1, a user control-interface 104may be physically packaged with the remote-controller 105. In otherembodiments, the user control-interface 104 may be packaged outside theremote-controller 105 and may communicate over a communication path(such as 114 or 115).

In embodiments where the user only needs to make “selections”, theuser-control-interface 104 may comprise a single “select” switch/button(e.g., 203) or control-input (e.g., single audio-command/sound). Exampleapplications include controlling televisions; DVD-players; cable-boxes;Digital-Video-Recorders (DVR's); home-theater-systems; music-systems;appliances; security systems; heating/cooling systems; thermostat;lights (on/off; dimming); closing shades; etc. The processing 106 may benotified [via communication path 114] whenever the user activates a“select” (e.g., “clicks” the select-button). FIGS. 2 a through 2 c showexamples of some possible locations for a physical “select” button 203.

In one embodiment, whenever the user first activates (e.g., “clicks” theselect-button or issues an audio-command):

-   -   the pointer-spot may be turned-on and/or    -   the control-display(s) may be automatically turned-on and/or    -   selection-menu(s) may be presented on the        active-control-display(s) 101.

Depending on the embodiment, the displayed selection-options may be adefault [e.g., high-level] menu or may be related to the current systemcontext. The pointer-beam 111 and/or the displayed selection-options maybe activated for only a pre-defined time [e.g., 5-20 seconds] after theuser has pressed the “select” button. The length of the predefinedtime-out may be independently optimized for each application and/orcontext. The pointer-beam 111 and/or active-control-display 101 [e.g.,menu-options] may be kept on, as long as the user continues to makeselections within the pre-defined time-out period.

In one embodiment, a valid “select” may be assumed to occur only whenthe user activates a “select” (e.g., “clicks” the select-button orissues an audio-command) while both the pointer-spot is turned-on andselection-options are available on the active/passive display(s). When avalid “select” request is made, the detectors and/or imager(s) maycapture one or more images of the pointer-spot 112 location. Forexample, the special case of initially pressing the “select” button toturn-on the pointer-beam and the control-display(s) 101 is notinterpreted as a control-action by the processing 106.

In an alternative embodiment, a multi-position-control switch (e.g.,203) may be used. In one embodiment, the three switch positions may be:

-   -   (1) pointer-beam and control-display are both off        (off-position).    -   (2) pointer-beam and display-menus are both activated; to allow        the user to move the pointer-spot to the desired        selection-option.    -   (3) the “select” position: pointer-beam on and the        detectors/imager(s) acquire one or more images of the        pointer-spot 112.

The three-position button/switch may be spring loaded so that itautomatically returns to the off-position whenever the user is notpressing on it. The three-position-button may provide the user withgreater control over when the control-selections are being displayed onthe active-control-display 101 [since the user is not bothered by theautomatic time-outs used in some of the other embodiments].

FIGS. 2 a and 2 b provide examples of the use of control-button(s) 203in hand-held remote-controllers 105. The control-button(s) 203 may bepositioned on the remote-controller 105 so that they may be activatedusing the thumb of the same hand that is holding the remote-controller105. FIG. 2 c shows an example of the use of control-button(s) 203 in awearable remote-controller 105.

The user-control-interface 104 may also be implemented to accomplish anydesired set or subset of computer mouse functions. Moving the cursor maybe easier than with a computer-mouse; since the cursor-location may beimmediately defined by the position of the pointer-spot when a “select”occurs. In addition [unlike a computer-mouse], a surface is not neededsince the user simply orients the remote-controller 105 to position thepointer-spot 112 at the desired cursor location.

Other equivalent computer-mouse functions may be optionally incorporatedinto the user-control-interface 104. For example, the equivalent of aleft mouse button may be accomplished by the “select” button of theuser-control-interface 104. Following a selection, the computersactive-location (e.g., a blinking symbol) may then be moved to (e.g.,indicated at) the location that was “selected” on the active-display101; or the “selected” object may be highlighted on the active-display101.

Similarly, the equivalent of a right mouse button [that may cause a setof menu-options to pop-up] may be accomplished with another button [orswitch position] 203 in the user-control-interface 104.

Scroll-wheel(s) similar/equivalent to those computer-mouse [e.g., tochange what is shown in a display window] may also be optionallyincluded in the user-control-interface. In one embodiment [supportingone handed operation], the buttons and scroll-wheels included in theuser-control-interface 104 may be positioned on the remote-controller105 so that they may be activated using the thumb of the same hand thatis holding the remote-controller 105.

The equivalent of a mouse drag-and-drop function may be accomplished byholding the “select” button down once the pointer-spot is at theobject-of-interest and then moving the pointer-spot to the new desiredlocation and then releasing the “select” button. The detectors/imager(s)may capture an image when the “select” button is initially pressed andagain when the “select” button is released [and perhaps at intervalsin-between]. Other computer mouse functions: includingselection-of-area; or selection-of-text or designation of thelocation-of-action, may also be similarly implemented. Distinguishingbetween a “single-click” and a “double-click” and between the“press-down” and “release” of a button(s) may also be useful in someembodiments.

In some embodiments, a keyboard or other display of alpha-numericcharacters and/or symbols [e.g., a virtual-keyboard] may be temporarilypresented on the active-display 101; so the user may select a sequenceof the symbols as a control “input”. For example, when in a channelselection mode; an active-display 101 may show a virtual-keyboard withthe numbers 0 thru 9. The user may then use the pointer-spot 112 toselect a sequence of numbers to go to a specific channel (e.g., select“3” then “5” then 1” to go to channel 351). The numbers may be shown intheir selected sequence on a portion of an active-display 101, so theuser can verify what is happening (e.g. “35_” may be displayed after thefirst two selections have been made).

In some embodiments, a special area may be (temporarily) placed on theactive-display 101 for the user to draw “input” symbols using thepointer-spot. For example, the user may draw alpha-numeric characters orsymbols with the pointer-spot 112 within a predefined area on theactive-display 101. A sequence of images containing the pointer-spot maybe captured by the detectors/imager(s) and processed to extract thepattern or shape that the user has drawn with the pointer-spot. Theprocessing 106 may then combine/connect the “spots” to determine thecharacter that the user had drawn out. Alternatively, thedetectors/imager(s) may use an increased integration-time so thepointer-spot is captured by [e.g., “smeared”] across many detectorelements that see the pointer-spot's path of movement.

In some embodiments, the user may issue commands by moving thepointer-spot to create a predefined pattern or symbol. For example, theuser may map out an “M” pattern with the pointer-spot to indicate apre-defined request (e.g, “M”=a specific-command).

In other alternative embodiment, portions of the user-control-interface104 may include voice-recognition or sound-recognition. For example, thecontrol-interface 104 may respond to words or phrases or sounds from auser. For example, the user may say “menu” and a control-menu may beautomatically shown on the active-display(s) 101 and the pointer-beammay be activated. After the user has placed the pointer-spot on thedesired selectable-object(s); the user may say “select”, which may thencause the detectors and/or imager(s) to acquire one or more images. Whenthe user says “done”, the pointer-beam and control-menus may beautomatically de-activated. The remote-controller 105 may have amicrophone to capture the sound; an analog-to-digital converter; andprocessing to recognize specific words or phrases or sounds (e.g.,verbal commands).

In another optional embodiment, the user-control-interface 104 may besimplified by including sensors within the remote-controller 105 todetect when the user is actively using the remote-controller 105. Forexample, the pointer 102 light-beam 111 may be automatically activatedwhenever sensor(s) embedded within the remote-controller sense that theuser is holding or using the remote-controller 105. This may eliminatethe need for a distinct pointer-beam-on and/or control-menu-on controlwithin the user-control-interface 104. Examples of sensors that may beincorporated into the remote-controller to automatically control thepointer-beam and control-menus include:

-   -   Heat or thermal sensor(s) to detect heat from the user's hand.    -   Motion sensor(s), such as accelerometer(s), to detect that the        user is moving the remote-controller 105.    -   Pressure sensor(s) to detect that the user is holding/orienting        the remote control.    -   In addition, the pointer-beam may be automatically deactivated        after a predefined time-period without detected user activity        [e.g., 20 seconds].

Any combination of these and other methods may be used for theuser-control-interface 104.

Pointer Implementations:

In general, any method of pointing at objects-of-interest that may beobservable by both the user and the detectors/imager(s) may be used.

In one embodiment, the pointer 102 is a light emitting device, capableof emitting light that will be easily visible to both the user and thedetectors/imagers either directly or when reflecting offobjects-of-interest.

In one embodiment, the pointer-spot may form a spatial pattern in theform of a small dot or disk shape when reflected off anobject-of-interest or other objects that are near an object-of-interest.

The pointers output power; wavelength; and beam focus may be chosen withconsideration of human eye safety and the safety of any objects that maybe illuminated by the pointer.

The illumination output from the pointer may be focused so that at alldistances from the pointer to an object-of-interest, the pointer-spot iseasily observed by both the user and the detectors/imager(s).

In one embodiment, a portion (or all) of the pointer's emissionwavelengths may be directly visible to the unaided human eye (e.g., inthe range of ˜0.4 to ˜0.8 um) and also detectable by thedetectors/imager(s). In another embodiment, the pointer-device'semission wavelength range is limited to a narrow range of wavelengths(e.g., red) within the visible region of the unaided human eye.

In one embodiment, the pointer 102 may utilize components similar tovisible the laser-pointers [e.g., a red laser-pointer] that widely usedto point at locations in viewgraphs, poster-boards, etc during apresentation.

In an alternative embodiment, the pointer-beam may contain both visiblelight (e.g., that the user can see) and non-visible light (e.g., thatthe detectors/imagers can see); where both are aligned to occur at thesame pointer-spot location 112. The wavelengths and bandwidth of thenon-visible light from the pointer-spot 112 may be chosen to minimizethe amount of stray light [e.g., ambient and/or emitted from thedisplay] that will reach the detectors/imager(s). An optical filter maybe placed before the detectors/imagers to only pass the band ofnon-visible light that is emitted from the pointer 102.

In another alternative embodiment, the pointer-beam's 111 wavelengthrange may be totally outside the visible range of the human eye (e.g.,in the non-visible wavelength ranges). A user operating a pointer 102,may wear special eyeglasses [or utilize other equipment] that re-mapsthe invisible pointer-spot 112 into wavelengths that the user can see.This may be useful for cases where a visible pointer-spot may interferewith other users that are also viewing the detection-region 921 [e.g.,the display region] with the naked eye. In this way, other users(without the proper equipment) will not be distracted by thepointer-spot 112.

In some embodiments or modes of operation, the pointer's light beam 111may be modulated or cycled in a known on-off pattern [possibly under thecontrol of the processing 106] to support capabilities such as:

-   -   Allow the user to see objects that are “underneath” (e.g.,        hidden-by) the pointer-spot; when the pointer-spot temporarily        shuts off.    -   Reduce pointer power usage (e.g., extend battery-life).    -   Distinguish between multiple pointers that may be simultaneously        operating.

In some alternative embodiments, one user's pointer-spot 112 may bedistinguishable from other user pointer-spots. Methods used (separatelyor in combination) to distinguish between multiple pointer-spots mayinclude:

-   -   Wireless communication (of an ID or codes) from the        remote-controller to the processing 106.    -   Variations of the pointer output under the control of the        processing 106.    -   Differences or variations in the pointer's wavelengths or        colors.    -   Modulation of the pointer wavelength, color or intensity of the        pointer's light output with time [e.g., pointer-unique on-off;        flashing; or pseudo-random pattern(s)].    -   Different spatial patterns or shapes of the pointer-spot 112        (e.g., circle, square, diamond or multiple-dot patterns).

In another alternative embodiment, the pointer 102 may project lightover the full field-of-view of the detector(s) and/or imager(s) exceptfor a small “spot without light” (e.g., a dot of “shadow/darkness”)where the “shadow-spot” represents the pointer-spot. The region aroundthe “shadow-spot” is illuminated by the pointer's light output. In anexample embodiment, the pointer's light output may be similar to aspotlight or flashlight except for a small “spot” without light(“shadow-spot) is created near the center of the pointer's light output.This may be useful for cases where no ambient light is available toilluminate the detection-region 921 [e.g., display region] where theuser is trying to locate objects-of-interest.

Detector and Imager Implementations:

The detectors and/or imager(s) may be capable of imaging thepointer-spot at any location within the pointer-spot detection-region921 [e.g., display region]. In some embodiments, area-array imager(s)similar to a still-image-camera or a video camera may be mounted orattached to an active-display 101 or a display-frame 941 so thereflection of the pointer-spot 112 may be detected anywhere within theregion of pointer-spot detection. In other embodiments, line imagers maybe used. In other embodiments, detectors may be distributed across thedetection-region 921 and/or across objects within the detection-region921. Details for each these approaches is discussed in the followingsections.

In some embodiments, the detectors and/or cameras may be activated tosense the pointer-spot 112 when initiated by a user action [e.g.,“select”] or when initiated by the processing 106. In some embodiments,the detectors and/or imager(s) may acquire a single image whenever theuser initiates a “select” at the user-control-interface 104. In someother embodiments or situations, the detectors and/or imager(s) may takemultiple images [e.g., when the user is moving the pointer-spot 112 to“select” a region or group-of-objects].

The number of detectors and/or cameras that are used may depend on

-   -   The dimensions of the detection-region 921.    -   The location accuracy of pointer-spot detection required by the        application [e.g., finer spatial resolution may require more        detectors].    -   Whether the display is curved; flat; 2-dimensional or        3-dimensional.    -   The acceptable thickness of the display along with the        connected/attached detectors/cameras.

In some embodiments, a band-pass optical filter that only passes certainwavelengths emitted by the pointer; may be placed in-front of thedetectors and/or imagers. For example, if a narrow-bandwidth red laserpointer is used in the remote-controller 105; then the optical filtermay pass only those red-light wavelengths that are emitted by thepointer. Use of the optical filter may:

-   -   Allow easier detection of the pointer-spot;    -   help to improve the detector signal-to-noise ratio; and    -   reduce false-detections by decreasing the amount of stray light        that may reach the detectors.

Viewing the Pointer-Spot Using Camera(s):

One or more cameras may be attached-to or integrated-into the structurearound the display region [e.g., a display-frame 941], so that thedetectors within the camera(s) may “see” the pointer-spot 112 as itreflects-off active-display(s) 101; fixed-display(s) 109; and/orreal-world-objects located within a display-region 921.

Note that although some of the following examples may show adetection-region 921 that is the same as an active-display 101; ingeneral any combination of active-display(s) 101; fixed-display(s) 109and/or real-world-object(s) may appear within the display region [e.g.,detection-region 921].

FIG. 8 a shows an example where two cameras (131, 132) areintegrated-into or are attached to the display-frame 941 of anactive-display 101. The cameras (131, 132) are located above thedisplay's surface so they will have a view of the pointer-spot when itreflects off an object in the display region. In some embodiments, tominimize the display-frame thickness, the cameras (131, 132) may belocated just above the display surface so each detector in a camera mayhave a shallow viewing angle. As shown in FIG. 8 a, each detector in acamera may have an angular-field-of-view of a slice of thedetection-region 921, where the size of the viewed region may bepie-piece shaped and may increase significantly with the distance fromthe cameras (131, 132). For example, one detector in camera 1 (131) mayhave a view of region 871 while one detector in camera 2 (132) may havea view of region 872. The region 875 in the display region may bedefined by the simultaneous view by one specific detector in camera 1(131) and one specific detector in camera 2 (132). When the pointer-spotis within region 875 of the display region, the reflected pointer-spotmay be simultaneously visible to one specific detector in camera 131 andone specific detector in camera 132. The camera optics and number ofdetectors in the cameras may be used to meet a desired spatialresolution [e.g., maximum size of a region 875] that is appropriate fora specific application. Depending on the camera height 862 and theoptics-design, the cameras (131, 132) may utilize either an area-arrayor a line-array of detectors. Line 821 in FIG. 8 a shows the location ofthe cross-sectional view that is illustrated in FIG. 8 b.

FIG. 8 b shows a cross-sectional view 821 (see FIG. 8 a) of thedisplay-frame 941 and the active-display surface 851 showing an examplewhere the cameras (131, 132) may be mounted on the upper portion (853,855) of the display-frame 941. In some embodiments, to minimize thethickness 861 of the display-frame 941, the cameras may be located asmall distance 862 above the display-surface 851, so that the camerasmay have a relatively shallow viewing-angle of the active-display 101. Aprotective cover 854 may surround the back and sides of theactive-display and be used to provide additional structural integrity.

Note that although FIGS. 8 a and 8 b illustrate an embodiment with anactive display type structure 852 [e.g., a liquid crystal displaypanel], in other embodiments images may be projected [e.g., projectiondisplays] into a surface [e.g., a wall or screen] in the display regionthat is defined by a detection region 921 and/or a display-frame 941.

FIG. 10 shows an example where three active-displays [101 a, 101 b, 101c] and a plurality of fixed-objects 109 [e.g., 421, 422, 423, 424] arelocated within a pointer-spot detection-region 921 [display region]defined by a detection region 921 or display-frame 941. For example, allof these displays and fixed-objects may be packaged together in acabinet or common assembly. In this example, two cameras (131, 132) mayview the pointer-spot 112 wherever it is placed on any of theactive-displays 101 and on any of the fixed-display 109 objects. Notethat any of the active displays 101 in these examples may be any type ofactive display 101 including projection type displays.

FIG. 9 a shows an example utilizing a line of detectors along the X-axisand Y-axis [971 and 972 respectively] that are attached to theactive-display 101 or display-frame 941. Optics in front of thedetectors may allow each detector to view a narrow slice [982, 983] ofthe pointer-spot detection-region 921. The area viewed [981] by both anX-axis and a Y-axis detector may be used to determine the location ofthe pointer-spot 112.

FIG. 9 b shows an example utilizing a single area-view camera that isattached to the active-display 101 or display-frame 941. The camera 991may be attached to the active-display 101 and/or display-frame 941. Theheight of the camera above the detection-region 921, may need to behigher than some of the other multi-camera embodiments, in-order toobtain sufficient spatial resolution [e.g., reduced image warping]. Insome embodiments, the camera(s) may be mounted on a camera-assembly thatis attached-to and/or pops-out from the display-frame 941, in a mannerthat allows calibration of the detector viewing-locations duringmanufacture. The camera assembly may be manually popped-out by the useronce during initial set-up or may be automatically deployed duringremote-control operation.

FIG. 9 c shows an example utilizing multiple cameras positioned alongboth the X and Y axis of the active-display 101 or display-frame 941.Each camera may have a view of a portion of the display region. Forexample, camera 2 (136) may have a view 135. Each detector in thecameras may view a portion of the detection-region 921. In someembodiments, the pointer-spot location may be determined by the commonarea that is viewed by an X-axis detector in cameras 135, 136 or 137 andby a Y-axis detector in cameras 138 or 139.

FIG. 9 d shows an example utilizing cameras (131,132,133,134) attachedat the four corners of the display-frame 941. In one embodiment, eachcamera may have an approximately 90 degree field-of-view (e.g., 135).Compared with FIG. 8 a, the additional cameras may be utilized toincrease the spatial resolution of pointer-spot within the displayregion [e.g., detection-region 921].

Reflected-Light from Pointer-Spot Versus Imager Viewing-Angle:

For embodiments that use a camera(s) to detect the pointer-spotreflection off objects, the amount of light that is reflected may varyas a function the viewing-angle of the camera(s). For flat surfaces, theamount of reflected pointer-spot light may decrease significantly whenthe imager(s) are looking at a small angle relative to the objectsurface. Irregularities in the display-screen or object surfaces may beused to improve pointer-spot visibility at such shallow viewing angles.Many types of active-displays naturally contain irregularities [e.g.,layers of films and pixel structures] that act as tiny reflectors thatwill disperse the pointer-spot toward the imager(s) and improvedetection of the pointer-spot at shallow viewing-angles.

In some embodiments, additional small structures that may act likeangle-reflectors may be embedded within or attached-to the displayscreen [during display manufacture or retrofitted later] to improvepointer-spot detection at shallow viewing-angles. The angle-reflectorstructure(s) may be designed to minimize the degradation of the normaldisplay image quality. In one alternative embodiment, a substrate [e.g.,a clear-plastic sheet] with very small embedded angle-reflectors may beattached to the display surface to improve pointer-spot visibility atshallow viewing angles. The angle reflector structures may be designedto optimize the reflection of pointer-spot wavelengths but to minimallyimpact the display wavelengths. For example, the reflector size[relative to pointer-spot wavelengths] and optical coatings/filters maybe employed. This may be more easily accomplished for embodiments wherethe pointer-spot includes non-visible wavelengths (in addition tovisible wavelengths).

Similarly, existing displays may be retro-fitted to become morereflective by attaching a substantially transparent substrate containingangle-reflectors [or other type reflectors] onto an existing display'ssurface. For example, a flexible clear plastic sheet containingangle-reflectors and with an adhesive backing may be affixed to thefront of an existing display to improve its reflectivity at shallowviewing angles. The impact on the viewed display sharpness may beminimized. Many existing displays may contain enough films/layers toreflect sufficient pointer-spot light to the imager(s) even at shallowviewing angles so that such special reflectors may not be needed.

Similarly, when needed, the roughness and surface characteristics offixed-objects and real-world-objects can be designed to allow improvedreflectivity of the pointer-spot at shallow viewing-angles.

Camera Implementations:

In one embodiment, the detectors may be embedded in one or more camerasthat are attached to the display-area and/or to a display-frame.Depending on the embodiment, area-array and/or line-array sensors may beused.

The cameras may be made both small and low power by utilizing integratedcircuit technology such as:

-   -   Line(s) and/or area-array(s) of light sensitive semiconductor        diodes and/or transistors.    -   Active Pixel Sensors (APS's) including those based on CMOS or        MOS semiconductor technology.    -   Line(s) or area-array(s) Charge Coupled Device(s) (CCD's).

Optics may be used to focus the image onto the image sensor(s). Theimager's focus-distance and field-of-view may be configured to allowimaging of the pointer-spot 112 at any location with the pointer-spotdetection-region 921.

Depending on the embodiment, the wavelength sensitivity range(s) of thedetectors/imager may be capable of detecting at least a portion of theillumination wavelengths of the pointer-spot as it is reflects offobjects in the detection-region 921.

Depending on the embodiment or mode, the detector/imagerillumination-sensitivity may be able to accommodate:

-   -   Variations in the intensity of the pointer-spot at differing        distances from the pointer 102 to the displayed objects.    -   Variations in the reflectivity of the pointer-spot with camera        viewing-angle and the when reflecting-off different types of        objects.    -   Detect modulations of the pointer's intensity in-order to        distinguish between multiple pointers (if used).

In some embodiments, the detectors/imagers may also include capabilitiesto prevent blooming due to the excessive reflected light frompointer-spot and/or “glint” from mirror-type reflection effects.

-   -   In some embodiments, only a single image may be acquired per        user action (e.g., “select”). In other embodiments, a sequence        of images may be acquired by the detectors/imager(s) based upon        user action(s). The sampling time(s) and sampling rate may be        automatically timed and/or sequenced and/or varied under the        control of the acquisition software and/or processing 106. In        other embodiments or modes, the imaging-rate (i.e., temporal        sampling rate) may be uniform and/or variable.

Depending on the embodiment or mode, the spatial resolution (e.g.,spatial sampling) of the detectors/imager(s) may be fine enough to:

-   -   Determine the pointers-spot location on the active-display 101        or fixed-display 109 with an acceptable precision.    -   Distinguish details of the reflected pointer-spot to uniquely        distinguish between pointers with different pointer-spot        patterns (when used).

Viewing the Pointer-Spot Using Distributed Detectors:

In another alternative embodiment, detectors capable of imaging thepointer-spot may be distributed across the pointer-spot detection region921. In the example embodiment shown in FIG. 9 e, detectors may bedistributed throughout the active-display-area 901 of an active display101 or a pointer-spot detection region 921. FIG. 9 f shows one detailedembodiment of the region 931 shown in FIG. 9 e. Note that in general thedistributed detectors may cover a portion of the display region[detection-region 921], while camera(s) are used to image other portionsof the display region [detection-region 921].

FIG. 9 f shows one example of a pattern of the detectors 951 that may bedistributed within the pointer-spot detection-region 921. Note that ingeneral, the detectors may be distributed across active-display(s) 101,or across fixed-display(s) 109 or across fixed-object(s) or acrossreal-world-object(s). The spacing between the detectors 951 may be madeless than the minimum pointer-spot size [e.g., the pointer-spot regionof illumination 112], so that the pointer-spot may be detected at anylocation within the pointer-spot detection region 921. The detectors 951(and their associated circuitry and interconnections not shown in thefigure) may be made small so that they may take up a very smallpercentage of the display-region and may not degrade the quality of thedisplayed-image the user experiences. The detectors 951 themselves maybe made as small as a quarter of a wavelength of the pointer-spot lightbeing detected, although they may be made larger to easemanufacturability. Many alternative detector patterns may be used andthe pattern may be chosen to ease manufacturability and minimizedegradation of the display's image-quality.

In one embodiment, the detectors and their associated circuitry andinterconnections (e.g., wiring) may be embedded in the active-display101 as part of the display manufacturing process.

Many display technologies [e.g., active-matrix LCD's; certainplasma-displays; surface-conduction electron-emitter displays;organic-LED displays; etc] utilize transistors [e.g., poly-silicon thinfilm transistors] and/or diodes to control the light-output of eachindividual pixel-element in the display. Additional dedicatedtransistors and/or diodes may be distributed across the display area todetect the presence of the pointer-spot. For example, the variation inleakage current caused by the pointer-spot's presence; may be integratedat a detector and may then be multiplexed and sensed. Where only thepresence or absence of the pointer-spot may need to be detected; a lowerquality of light-sensitive transistor and/or diode and readout sensingcircuitry may be used. The added detectors may be interconnected usingthe same method of connecting the individual display-pixel-elements suchas Indium Tin Oxide (ITO) and may be fabricated to be mostly transparentso as to not compromise the quality of the display.

In an alternative embodiment, the detectors and their associatedcircuitry and associated inter-connections may be fabricated on aseparate substrate (e.g., silicon, sapphire, glass-plate, clearplastic-sheet, etc) that is then attached to the display's pointer-spotdetection-region 921. There are many semiconductor processes, known tothose skilled in the art, that may be utilized to create distributeddetectors on a flexible (e.g., thin) sheet of substrate material. Insome embodiments, an optical filter and/or an optical lens may be placedin front of each detector during the manufacturing process. In oneembodiment, the substrate may be a flexible sheet or roll containingdistributed detectors [and their associated circuitry]. The sheet ofdetectors and associated circuitry may have an adhesive backing to allowits attachment to active-displays 101 and/or fixed-displays 109 and/orreal-world-objects. Such a sheet may also be used to retro-fit olderactive displays 101.

Retro-Fitting Older Displays with a Display-Frame:

An alternative embodiment of the invention may allow retro-fitting theinvention to older displays that do not have detectors and/or cameras. Aseparate structure [e.g., a display-frame 941] may be manufactured andcalibrated with attached/embedded detectors and/or cameras. The user maypurchase the display-frame 941 assembly and then attach the structure toan older active-display 101.

In another alternative embodiment, the size of the display-frame 941[e.g., structure] may be made adjustable, so that a single frame designmay be retro-fitted to a range of displays of different sizes and/oraspect-ratios. For example, the X and/or Y dimensions of thedetection-region 921 may be defined by telescoping segments that siderelative to each other to allow the display frame to be expanded orcontracted to accommodate a wide variety of different sized displays.Once the detection-region 921 has been adjusted to a desired display,the frames telescoping sections may then be locked into place. Locatingthe cameras in the corners of an adjustable frame [as shown in FIG. 1,8, 9 d, or 10] enables an adjustable-frame to be used with a wide rangeof detection-region sizes and aspect-ratios. For the two cameraconfigurations shown in FIGS. 1 and 8 a, the display-frame may beconfigured as a telescoping-bar [rather than a rectangular frame] withcameras at each end of the bar. The bar may then be attached to theupper X-axis portion of the display-frame 941 [e.g., above the displayregion]. During initial set-up of the display-frame, the processing 106may obtain configuration-data that corresponds to the particulardimensions the frame has been adjusted to.

Active-Display Implementations:

An active display 101, has the capability to change what is appearing onits display. Examples of active-displays 101 include cathode ray tubes(CRT); liquid crystal displays (LCD); plasma-displays;surface-conduction electron-emitter displays (SED); digital lightProcessing (DLP) micro-mirror projectors/displays; front-side orback-side projection displays (e.g., projection-TV); projection ofimages onto a wall or screen; computer-driven projectors;digital-projectors; televisions; DVD-player-screens; light emittingdiode (LED) displays; variable 3-D displays; variable holographicdisplays; or any other type of display where what is being displayed canbe changed based on context and/or user actions. In general, activedisplay(s) of any type may be used as all or part of a pointer-spotdetection-region 921.

The images that are displayed on the active-display(s) 101 may becontrolled by the processing 106. In some embodiments, theactive-display(s) 101 may show one or more control-menus or images withselectable-objects and the user may then use the pointer-spot toindicate their desired selection. An active-display 101 may displayimages of selectable-objects, selection-options, control-selections ormenu-options, that a user may select or interact-with.

In some embodiments, multiple active-displays 101 may be activesimultaneously and the processing 106 may coordinate activity across orbetween the multiple displays. In some embodiments, a singleactive-display 101 may be dedicated to the remote control function andmay be used to control one or even all of the user's electronic-devices.In an alternative embodiment, multiple control-active-displays are madeavailable at many locations and the user may control all theirelectronic-devices from any one of the displays they happen to be near.

In other embodiments, the active-display(s) 101 may be embedded withinone or more of the electronic-devices being controlled and may betemporarily utilized to perform the remote control functions. Forexample, a television-display [or DVD player display; or homemedia-center display] may be temporarily used as theremote-control-display, when needed, in-order to temporarily displayimages of selectable-object(s)/selection-menus that control otherelectronic-devices that don't have their own display [such as a stereosystem; Compact-Disk (CD) player; iPod player; Video Cassette Recorder(VCR); Digital Video Disk (DVD) player; Digital Video Recorder (DVR) andcable-channel box].

In other embodiments, the control function and active-display(s) 101 areintegral to the electronic-device. For example, in a personal computer,the remote-controller 105 may be perform the same functions as a mouse(or any other type of computer pointing device) by acting upon orinteracting-with selectable-objects that are displayed on the computerdisplay.

Menus, Selectable-Objects and Point-of-Action:

The user may use the pointer-spot 112 to select and interact withobjects or groups-of-objects that are within the detection-region(s)921. In some embodiments, the processing 106 has knowledge of the eachobjects location and the allowed types of selection/interaction with theobject.

Selectable-objects may appear on active-display(s) 101 and/orfixed-display(s) 109 and/or as real-world-objects. Typically,fixed-displays 109 and fixed-objects are unable to change theirappearance. In general, selectable-objects may be two-dimensional (2D)or three-dimensional (3D).

In some embodiments, the menu-images and/or selectable-objects that areshown on the active-display(s) 101 may change over time as managed bythe processing 106 and may be based upon the user actions; the operatingstate and/or the current-context. To reduce user complexity, only thoseselection-options that apply in the current context may be presented tothe user at any given time. For example at a given time, the user may bepresented with only that subset of control buttons that applies in thecurrent context; rather than being presented with all the controlbuttons that appear on prior-art remote-controllers.

In some embodiments, real-world-objects may be within thedetection-region 921 and may not change with context or usercontrol-actions. Examples of fixed real-world-objects include mechanicalcontrols or switches that control a device (e.g, a physical on/offbutton on the front of a device); hardcopy text and/or symbols; printedmaterial (e.g., control icons/images or text printed onpaper/poster-boards); or any other type of fixed-object. The processing106 may be provided with configuration data that defines the location ofall the selectable fixed-objects within the detection-region 921 anddefines the corresponding actions to take when selected or interactedwith.

In some embodiments, only a subset of the objects, in an active-display101 or passive-display 109, may be selectable, since some objects in thedisplay(s) may only be present for ornamentation or artistic purposes(and are not intended to be selectable). As an example, a DVD menu-imageon an active-display 101 may contain a number of selectable objectsalong with non-selectable objects and possibly underlying non-selectablebackground stills/image(s) from a movie.

Examples of what the selectable-objects can represent are:

-   -   “Setup or configuration” options such as “input source”;        “language choices”; “audio-mode”; volume; channel; etc    -   Content choices the user may view such as movies; movie scenes;        DVD special features; etc.    -   Menu control options such as “Show more options”; “exit (from        current command mode)”; “exit and close menu”; “back”;        “forward”;    -   “undo”; redo” configuration actions.    -   Web browser objects.    -   Any other type of configuration or control or choice or        selection.

FIG. 4 a shows an example of selectable-objects that may be shown on anactive-display 101. The selectable-objects may be any type ofdisplayed-object such as text; icons; or images. For example, the usermay be presented with a set of control-objects that the user may selectfrom. For example in FIG. 4 a, the user may select one of thecontrol-objects shown on the active-display 101: “TV” 401; “DVD” 402;“Music” 403; “Exit” 404. For example, when the user places the pointerspot is on/near the “DVD” selectable-object and activates a “select”, amenu of options for controlling the DVD-player may be then be displayedon the active-display 101.

As also shown in FIG. 4 a, the active-display 101 may also containselectable-objects that control and/or configure a device. For example,when the pointer spot is on the up-arrow 411 or down-arrow 412 and the“select” is activated, then the device's current channel may be change.Similarly, when the pointer-spot is on the up-arrow 423 or down-arrow424 and the “select” is activated and then the device's volume may bechanged. In some embodiments, the menu-selections may only temporarilyutilize a small portion of the active-display or may be transparent sothe underlying images are still visible. In alternative embodiments, adedicated portion of an active-display 101 may display the currentlyneeded menus-selections.

FIG. 4 b shows an example where both an active-display 101 and afixed-display 109 containing selectable fixed-objects (421; 422; 423;424; 425; 426; 427; 431; 432) are located within the pointer-spotdetection-region 921. The fixed-objects may be any type ofreal-world-object including the actual physical controls of a device.For example, the channel-up-button 421 and channel-down-button 422(e.g., physical hardware buttons) located next to the active-display 101may be selectable-objects. For example, if the user places thepointer-spot 112 on the channel-up-button 421 and activates a “select”;then the detectors may capture the location of the pointer-spot 112 onthe fixed-object; and the processing 106 may then complete the user'srequest, which is to move one channel-up. Note that the actualhardware-button is not pressed but command-codes equivalent to thehardware-button function may communicated 115 to the specific devicebeing controlled 108. In an alternative example of FIG. 4 b, all theobjects in the fixed-display 109 may appear [e.g., were printed] on astructure [e.g., a plastic surface] that is located within thedetection-region 921. For example, frequently used controls (e.g.,volume) may be printed as fixed-display-objects around an active-display101 but within the detection-region 921.

In embodiments that perform the equivalent-functions of a computer-mouseor pointing-device, the user may use the pointer-spot to definepoints-of-action or areas-of-action that may appear in active-display(s)101. For example, in an interactive drawing program on a personalcomputer, all or substantially-all of the objects may be selectable orinteractive. For example, the user may interact with selectable objectsin-order to select, cut and paste objects; or modify objects such asstretching or changing the object dimensions. The currently activecursor location or point-of-action or area-of-action may be highlightedon the active-display 101.

In situations or embodiments where the processing is unable to obtainknowledge of the locations [and/or the control-meaning] of the objectsin a control-image, virtual navigation buttons/icons [up; down; left;right] may be presented to the user on an active-display 101. In somecases, the virtual navigation buttons may be (partially) transparent sothat the underlying image(s) is still visible to the user. For example,if the location information is unavailable for the special featuresnavigation of a DVD, the equivalent navigation-buttons available on aprior-art remote-controller may be presented virtually to the user toallow navigation of the DVD's content.

The following considerations related to active-displays, fixed-displaysand selectable-objects may be helpful in certain embodiments:

-   -   Displays and objects may be designed to make the pointer-spot        easier to detect.    -   The difference between selectable and non-selectable objects in        display-images should be immediately apparent to the user;        in-order to minimize wasting the user's time by trying to select        a non-selectable object on the display.    -   Menu-items may be designed to minimize the amount of the        underlying entertainment or program that is masked.        Menu-selections may be partially transparent so the underlying        entertainment or program is still visible.    -   Menu-selections may be placed on part or all of the        active-display(s) 101.    -   Menu-items and control-screens may be automatically removed from        an active-display 101 when there is no user activity for a        certain time (e.g., 5 seconds).    -   The pointer-spot may blink on and off so the user can still see        display details that might be masked by the pointer-spot 112.    -   A user selectable-object (e.g., “help” icon) may be selected by        the user to see additional information. A pop-up box with        additional information may be optionally provided. Or a “help”        option accessible via a “right-mouse-button” equivalent.    -   “Undo” and “Redo” buttons may be included to enable a user to        easily recover from accidental configuration mistakes/errors.

Processing Implementations:

The processing 106 may include circuitry to allow communication 114 withthe remote controller 105 and allow communication 115 with theactive-display(s) 101 and the device(s) being controlled 108. Forexample, captured-images may be transferred to the processing 106in-order to determine the location(s) the user had selected and whichobject(s) the user is selecting or interacting-with.

The processing 106 may be in a single location or distributed acrossmultiple physical locations (e.g., enclosures or devices) that areconnected via a communication paths (114 or 115) such as a network. Forexample, the processing may be located with or within or distributedacross: the remote-controller 105 or active-display(s) 101 or theelectronic-device(s) being controlled or may be in a separateenclosure(s). In some embodiments, the processing 106 may be shared withthe devices-being-controlled 108.

The processing 106 may be implemented by any combination of: a computeror general-purpose-processor(s) capable of executing software programs(e.g., microprocessors); Digital Signal Processor(s) (DSP); a dedicatedhardware processing device; integrated-circuit(s) [IC's]; ApplicationSpecific Integrated Circuits[(ASIC's] or Field Programmable Gate Arrays[FPGA's]; or any other type of other electric/electronic circuitry.

The processing may be embodied as a program for execution by a processorsuch as a computer or DSP, and as a storage medium storing such aprogram. Further, the processing may be implemented as a method,apparatus, program or storage medium for separate application to aserver [or client] on a communication network.

Processing Functions and Approaches:

Depending on the specific embodiment, the processing 106 may perform oneor more functions including:

-   -   Maintain knowledge of the devices 108 that are connected to a        user's system.    -   Control and maintain knowledge of what is currently being        displayed on each of one or more active-displays 101 including        knowledge of all the selectable-objects.    -   Maintain a database of selection-menus and control-menus for        each device.    -   Have knowledge of all the fixed-display(s) 109 and real-world        objects that are within all the pointer-spot detection-region(s)        921 in the user's system. Maintain knowledge of the location(s)        of fixed-objects (and their corresponding control functions)        previously defined to be associated with a particular        electronic-device.    -   Control [and adjust] the operation of the pointer 102 and/or        detectors/imager(s).    -   Access or receive database; menu; and/or control information        from media inserted in user-devices being controlled (e.g., DVD        menu information).    -   Determine the location(s) and object(s) the pointer-spot        designated.    -   Determine which of several devices or displays, the        captured-image(s) and user-control actions correspond to.    -   Change the state or mode of the appropriate device based on user        selections and/or control-actions.    -   Change what is shown on the active-display 101 based on user        “selections”, control-actions and/or the current context.    -   Maintain a pre-programmed sequence of display images and/or        selectable-objects that are dependent on the ordering of user        selections.    -   In some embodiments, determining which one of a plurality of        detection-regions 921 is currently being used.    -   Other functions that are disclosed in this specification.

FIG. 12 shows one embodiment of a processing flow chart for determiningand performing a user's intended action. In block 1201, the processing106 may “identify the detector(s) viewing” the pointer-spot”. Forexample, the relative magnitude of the detectors may be examined todetermine the specific detectors that are detecting the pointer-spot112. In some embodiments (e.g., where the pointer spot has a knownintensity profile and the pointer-spot extends over multiple pixels),centroid-processing may be used to determine the “pointer-spot” locationwith greater precision (e.g., to an error that is smaller than the sizeof the pointer-spot). In many embodiments, this extra precision may beunnecessary.

For embodiments using cameras mounted near the display edge (e.g., FIG.8 a), each detector may see an angle-of-view. The length-dimension of anangle-of-view [e.g., the pixel length] may increase linearly with thedistance away from the camera as shown in FIG. 8. The resolution-size ofa camera pixel near the camera may be several times smaller comparedwith one the far side of the detection-region. For this reason, apointer-spot near a camera may appear in many more detectors; than apointer-spot that is located away from the camera. When multipledetectors are illuminated by the pointer-spot, the accuracy of itslocation may be improved by assuming the pointer-spot is near the centerof the illuminated detectors or by using centroid-processing techniques[e.g., using the relative detector illuminations along with the knownillumination shape of the pointer-spot].

In block 1202, the processing 106 may “determine the X:Y location of thecenter of the pointer-spot in the detection-region 921 or active-display101”. A “mapping of each detector or set-of-detectors to an X:Y displaylocation” 1222 may be provided as configuration-data. FIG. 7 b shows asimplified example of such a mapping.

In block 1203, the processing 106 may “determine whichselectable-object(s) or point-of-action the user designated with thepointer-spot”. The processing 106 may utilize a “mapping of the X:Ylocations in each display-image to a selectable-object” 1223. FIG. 7 bshows a simplified example of such a mapping. The display-images mayinclude control-menus; selection-options; and/or an interactive-images[e.g., for an interactive drawing or word-processing application]. Forsome display-images, some X:Y locations may not be associated with anyselectable-objects [e.g., are white-space or a non-selectable region].For embodiments that are similar/equivalent to a computer-mouse, thisprocess may also include establishing a new cursor location ordetermining the point/area in a computer display that is “selected”.

Each “selection” menu may have known locations for its selectableobjects. A mapping or lookup table can be pre-created for eachmenu-image which translates each display location to one of theselectable-objects or a non-selectable region.

In some embodiments, multiple menu-images may share the same mappingtable when all their selectable-objects are at the same locations. Tablesizes may also be reduced in some embodiments by sequencing throughmultiple levels of mapping tables.

If the processing determines that “no valid object was selected” inblock 1203, then the user may be notified of the selection error inblock 1204.

Per in block 1205, if the processing determines that “a valid object wasselected” in block 1203, then the processing 106 may perform the actionthat corresponds to the selected-object(s) and update theactive-display(s) with new display-image(s). The processing may utilize“database(s) or table(s) of actions-to-perform for eachselectable-object that appears in each display-image” 1225. FIG. 7 ashows a simplified example of such a table.

Once the processing 106 has determined the selected-object orpoint-of-action, then action(s) may be taken that correspond to that theuser's selection. Configuration-data 630 may be provided to theprocessing 106 to facilitate this. This configuration-data may include adatabase or tables that define what actions to take for eachselected-object in each control-image/menu.

In some embodiments, the selected-object [or theselected-point-of-action] may be the highlighted on an active-display101 in a way that also allows the user to simultaneously visually “see”that their intended selection has occurred. For example, theselected-object may be “highlighted” using a unique color; brightness;shading; framing; underlining; or modulation/blinking pattern.

Some examples of the types of actions that may be taken include:

-   -   Control-actions on one or more electronic-devices being        controlled (e.g., “change the channel” or “change the source”).        Control-actions may be communicated to the specific        electronic-device(s) 108 being controlled over a communication        path 115.    -   Displaying new image(s) or modifying the prior image(s) that        appear on one or more active-displays 101. New display-images        may include control-menus; selection-options; and/or an        interactive-images [e.g., for an interactive drawing or        word-processing application].    -   Highlighting or modifying the selected-object(s). This may        include taking actions that modify the selected-object.    -   Displaying pop-up menus; information overlays; or additional        details associated with the selected-object.    -   Any type of action equivalent to a computer mouse or other        computer-pointing-device used to control or interact with a        computer or other electronic device.

The actions to be taken may be defined in advance (e.g.,configuration-data) using table(s); database(s); flowcharts; conditionalevent sequences; state diagrams; conditional programming; or any othermethod.

FIG. 7 b shows part of a simplified table that may be used to map apointer-spot location to a particular selectable-objects present in thedisplay-region. For each control-menu (e.g., control-menu 5), aselection-region may be is defined for each selectable-object that is inthat menu-image. In this example, the second column of FIG. 7 b definesthe X:Y coordinates of the four corners of the rectangularselection-region that define the selection region for eachselectable-object in the display-region. More generally, theselection-region for each object may be defined to have any desiredshape.

FIG. 7 a shows a part of a simplified table that may be used to map fromeach selectable-object to the specific control-action(s) that areassociated with each selectable-object in each control-menu image. Thisexample may be typical of the remote controlling a television or aDVD-player or similar electronic-device. For example, per the second rowof FIG. 7 a, if “control-menu 5” is being displayed and the user“selected” the “up-volume” selectable-object, then “the volume isincreased by 1 increment” and “control-menu 5” is presented to the useragain so the user may make additional “control-type” selects that areavailable on that image-image. Entries may be provided for allmenu-images that contain selectable-objects and for allselectable-objects that are in each menu-image.

Design and Calibration-Data:

The mapping from each detector to a corresponding X:Y location withinthe detection-region 921 may be determined from the design-models and/orexperimentally with calibration procedures performed duringdesign/manufacture. For example, a calibration-grid of known X:Ylocations may be defined throughout a detection-region 921. Whencalibrating active displays, each grid-point may be illuminated on anactive-display 101 to define where a pointer-spot 112 will be placed.Once the pointer-spot has been placed at a known grid X:Y location, thedetector or detector(s) that are activated by the pointer-spot may berecorded. The number of points in the calibration-grid may depend of theaccuracy requirements needed for each particular embodiment. The designor calibration data may be provided to the processing 106 asconfiguration-data, during manufacture or initial set-up.

To speed up the calibration process, multiple X:Y locations that areappropriately spaced, may be simultaneously illuminated with multiplepointer-spots. A sequence of such multiple simultaneous locations may beused until a complete grid of X:Y locations has been experimentallytested.

Configuration-Data:

In some embodiments, configuration-data may be provided to theprocessing 106 to allow the remote-controller 105 to be interoperablewith any number of user-devices. Some examples of configuration-datainclude:

-   -   The model-number or other ID of each device being controlled.    -   A set of control-menus [for each device] that may be displayed        on an active-display(s) 101.    -   Definitions of the selectable-objects in each control-menu.    -   Definitions of the control actions that may be performed on a        user-device when each selectable-object is “selected”.    -   Definitions of the next control-menu or other image-content to        display on the active-display when each selectable-object is        selected.    -   Definitions of the control actions that are to be perform on a        user-device when each fixed-object is “selected”.    -   For entertainment media (e.g., DVD movie), definitions of the        menu-selections for each menu-image included on the DVD.    -   The communication path (e.g., RF channel) and the specific        control encoding necessary to control each of the user's        electronic-devices.    -   The infrared patterns/codes needed to control each of the user's        older infrared controllable electronic-devices.

In some embodiments, the configuration data that is needed for auser-device, may be provided to the processing 106 during the initialset-up of that user-device. For example, when a user adds a tenth device108 to their home-system, the configuration-data for thatelectronic-device may be input into the processing 106 as part of theinitial set-up so that the same remote-controller may be used to controlall 10 devices. The configuration-data may be saved into a non-volatilememory so set-up is required only once per device.

There are many ways that the configuration-data may be provided to theprocessing 106; including:

-   -   The configuration-data is provided by the        device(s)-being-controlled 108 over a communication-path 115.    -   The device-being-controlled 108 may have a media-drive (e.g., a        DVD or CD) that may accept media containing configuration-data.    -   The configuration-data is provided from the Internet and/or        other outside sources of information via a communication or        Internet/Network 118 path.    -   The configuration-data is provided via a media drive 117 (e.g.,        a DVD or CD) or interface that is attached to the processing.

In one embodiment, when the user adds a new device, the processing 106may access the Internet 118 to obtain the configuration-data needed toremotely-control or interact-with the new device.

Configuration-data may also be used to allow the remote-controller toact as a universal remote-control and/or to providebackward-compatibility with older infrared-controlledelectronic-devices.

Communication Paths and Plug-and-Play:

The electronic-devices-being-controlled 108 may include circuitry toallow communication 114 with the remote controller 105 and/or allowcommunication 115 with the processing 106 and the active-display(s) 101.

The communication paths 114 and 115 may be implemented using anycombination of communication technologies including both wired andwireless. Typical examples include coaxial cables; twisted pair cables;Ethernet networks; communication busses; Electro-optical (EO);fiber-optics cables; electro-magnetic (EM); infrared (IR);Radio-Frequency (RF); WiFi; Bluetooth; cell phone networks; and theInternet. Any combination of network(s) and point-to-point communicationapproaches may be used.

In one particular embodiment, the communication path 114 between theremote-controller 105 and the processing 106; is implemented using anon-line-of-sight wireless channel(s) (e.g., wirelessradio-frequencies); so that the remote-controller does not have to bewithin line-of-sight with the processing 106. Similarly,non-line-of-sight wireless channel(s) (e.g., wireless radio-frequencies)may be used for communication path 115.

In one particular configuration, the remote-controller 105 maycommunicate to an interface [e.g., receiver/transmitter] located at thedisplay region 921 which then communicates to the processing 106. Thismay assure a more line-of-sight communication path 114 to/from theremote-controller 106.

In another alternative embodiment, portions of the communication paths[114 and/or 115] may utilize the Internet or other Wide-Area-Network[e.g., cell-phone network] to allow devices to be remotely controlledfrom other locations [e.g., home devices controlled from work or anotherlocation].

In some embodiments, a non-direct line-of-sight wireless communicationnetwork (similar to Bluetooth or WiFi) with plug-and-play capability maybe used for communication paths 114 and/or 115. This may allow theelectronic-devices being controlled 108 to be located anywhere(including out-of-sight) and may reduce the cabling between theelectronic-devices. In addition, the electronic-devices may be easilyadded and removed from the communication network(s) using wirelessplug-and-play techniques.

In some embodiments, portions of the communication paths [114 and/or115] my have a plug-and-play or automatic device-discovery capability.When the user adds or removes remote-controllers and/ordevices-being-controlled to their system, the change is automaticallydetected by the communication path [114 and/or 115] and the processor isnotified so appropriate action can be automatically performed withminimal/no user action. There are many communication and networkstandards [e.g., Ethernet; WiFi; Bluetooth; Universal Serial Bus (USB);Wireless-USB; Power-line-networks] that support automatic discoveryand/or plug-and-play capabilities.

FIG. 11 shows a flow diagram for one embodiment of the automaticdiscovery and set-up [plug-and-play] of a new electronic-device that theuser has added to their system. Depending on the embodiment, any givenfunctional block may be: optional or performed in a different order;and/or multiple blocks may be combined into one block.

In block 1100, the user powers-up the electronic-device and connects theelectronic-device to the communication path 115 [such as a WiFi orEthernet network]. In some embodiments [e.g., wireless WiFi], theconnections may be pre-configured to automatically occur withoutrequiring any user action.

In block 1101, the communication path 115 automatically discovers theadded electronic-device that is now on the communication path 115.

In block 1102, the processing 106 may obtain information about the addeddevice. The information obtained by the processing 106 may include datasuch as the device-model; device-ID; device-serial-number; presentdevice-status and settings.

In block 1103, the processing may then obtain information such as set-upinformation; user-help-information; and set-up menus for theelectronic-device being added into the user's system. Some of thisinformation may be obtained from the electronic-device itself over thecommunications path 115. In some embodiments, the processing 106 mayobtain this information or updated information over the Internet [orexternal network] communication path 118. In some cases, information maybe obtained from media-drive(s) 117 attached to the processing or frommedia-drive(s) attached to the electronic-device being added.

In block 1104, the processing 106 may then obtain and configurecontrol-menus; operational-menus and user-help-menus for the addedelectronic-device. For cases involving older infrared-code controlledelectronic-devices, the processing may acquire the infraredsignaling-codes used for controlling the device.

In some embodiments, the processing may also obtain and configurecontrol-menus that reflect a combination of the user systems. Forexample, one menu may show a selectable listing of all the devices thatthe user can presently control. Or all the different TV channels orprogramming sources [satellite; cable; airwaves; Internet] that arecurrently available to the system, may be organized together on acombined control-menu(s). Or all the volume controls on the differentdevices may be coordinated through a combined volume menu.

In block 1105, the processing 106 configures the added device and theother system devices as necessary.

In block 1106, the user may be notified that setup is complete and theuser may operate and control the added device using the display menus.

In some embodiments, it may be desirable to lockout rogueremote-controllers, by limiting remote-control to only particularremote-controller(s). The processing 106 may be configured to onlyaccept communication and commands from remote-controllers that knowpredefined access-codes or identifier-codes/passwords.

In some embodiments, encryption (e.g., symmetric-key and/or public-keyencryption) may be used to secure communication contents across thecommunication paths 114 and/or 115.

Functional Example

FIG. 3 illustrates a simple example of the functional blocks that may beused to remotely control or interact-with any electronic-device that maybe controlled by selecting displayed menu-options or selectingfixed-objects. Examples of such devices include televisions;DVD-players; Digital-Video-Recorders (DVR); cable-television-boxes;home-theatre-systems; appliances; or heating/cooling systems. In asimple embodiment, the user may place the pointer-spot 112 on aselectable-object in a display region [e.g., detection-region 921] andthen press the single button 203 [e.g., “select”] that is on theremote-controller 105, to cause desired control-action to occur.

This example assumes a remote-controller 105 with a pointer 102 thatoutputs a light-beam 111. In some embodiments, configuration-data (asdiscussed elsewhere) may be used in one or more of the blocks shown inFIG. 3. In some embodiments, a one time setup may be performed for eachuser-device in-order to identify (e.g., model number) and to provide theneeded configuration-data to the processing 106, prior to controllingthe electronic-device.

As shown in block 300 of FIG. 3, the user holds the remote-controllerand the pointer's 102 output light-beam 111 is activated. The light-beam111 may be activated using many different embodiments such as

-   -   the user pressing a control-button 203    -   the user speaking a command (e.g., “beam-on”)    -   sensor(s) in the remote-controller 105 may automatically detect        that the user is actively using the remote-controller 105.

In block 301, selectable-object(s) [e.g., selection-options ormenu-items] appropriate to the current context may be presented to theuser on active-display(s) 101. In addition, other fixed-display(s) 109or fixed-objects may also be in the detection-region 921 and may also beavailable for selection by the user.

In block 302, the user moves the pointer-spot on or near the desiredselectable-object(s) on the active-display(s) 101 or fixed-display(s)109; and the user initiates a “select”. A “select” may be initiated invarious ways including:

-   -   The user presses a control-button 203 to signify “select”.    -   The user may release a control-button to signify “select”.    -   The user speaks a certain word or phrase (e.g., “select”).

The occurrence of the user activating the “select” may be signaled tothe processing 106 via communication path 114. In one particularembodiment, the communication path 114 may be accomplished with anon-line-of-sight wireless path; so the remote-controller 105 may beunconstrained by cords/wires.

In block 303, initiation of the “select” may cause the detectors and/orimager(s) to capture one or more images [capture detector sense data] ofthe pointer-spot.

In block 304, the image(s) [e.g., sensed detector data] is analyzed bythe processing 106 to determine the location of the pointer-spot withinthe detection-region 921 and to determine the selectable-object(s) thepointer-spot was on or near when the “select” was made.

If the analysis in block 304 indicates that the pointer-spot was not onor acceptably close to one of the known selectable-object(s), then theuser may be notified that no valid selectable-object(s) was selectedwith the pointer-spot (block 311). The user may be notified in variousways including:

-   -   A visual indication on the active-display 101.    -   An audio message (e.g., a voice message) and/or a sound (e.g., a        beep).

The user may try again by going to block 300.

If the analysis succeeds in identifying specific selectable-object(s)then user may be optionally notified of the success (block 305). Theuser may be notified of the action taken in various ways including:

-   -   Make the selected-object(s) into a “Highlighted-object(s)” on        the display.    -   Moving the point-of-action [e.g., cursor location] or        active-location (e.g., location of activity) on the        active-display 101 to the selected-location.    -   Providing another visual indication on the active-display 101.    -   Providing an audio message or sound.

In block 306, the control-action(s) that correspond to theselected-option is communicated to the device being controlled. A fewexamples of control-actions are:

-   -   Increasing or decreasing the volume.    -   Changing the channel up or down or going to a selected-channel.    -   The selected video or music selection may be started.    -   The next (forward) or previous (back) item in the playback        sequence may be started.

In block 307 as appropriate to user's current context, various items maybe now shown on the active-display 101. This may include:

-   -   Highlighting [perhaps temporarily] the object the user selected        on the display.    -   A temporary visual indication of the action taken [e.g., a        symbol showing that the volume was increased].    -   Different selectable-object(s) [menu-items and/or        selection-option(s)] may be presented to the user on the        active-display 101.    -   Removing all selectable-object(s) [menu-items and/or        selection-option(s)] from the active-display 101 (e.g., when all        selections have been completed).

The user may now make additional selections or take additional controlactions by continuing at block 300.

In some embodiments, the selection-menu(s) [e.g., control-menus orselectable-object(s)] options may be automatically removed from theactive-display 101 when the user does not take any action after adefined period of time. For example, the selection-menu(s) may beautomatically removed from the active-display 101 after 5 seconds ofinactivity.

Backward Compatibility with Existing Infrared Remote Controls:

In an optional enhancement, this invention may be implemented to alsocontrol older infrared-controlled electronic-devices. Universalcompatibility and/or backward compatible with old infrared user devicesmay facilitate the adoption of this invention.

In an alternative embodiment shown in FIG. 13, an RE-to-InfraredConverter 1300 may be attached to an older infrared-controlled-device,so that infrared-command-codes 1304 from an infrared-emitter 1303 willreach the infrared-receiver 1311 in the older device. The RE-to-InfraredConverter 1300 may contain an RF-interface 1302 to receive commands [andsend status back] over an RF-communication path 1301 with the processing106. The RF-interface may contain a radio-frequency receiver/transmitterthat allows non-line-of sight communication from the processing 106to/from the RE-to-Infrared Converter 1300. In some embodiments, theRF-communication-path 1301 may be the same as communication paths 115 or114 shown in FIG. 1. The RE-to-Infrared Converter 1300 may also containa battery 1305 so that no external wired-connections are need. Byattaching an RE-to-Infrared-converter 1300, an older device is no longerconstrained to be within infrared-line-of-sight of the remote-controller105 or the processing 106. This may allow some older infrared-controlleddevices [e.g., VCR and DVD players] to be placed out of sight of theuser.

The processing 106 may receive configuration-data from themedia-drive(s) 117 and/or Internet/Network 118 and/orDevice(s)-Being-Controller 108 that includes all the necessaryinformation for backward compatibility. The configuration-data mayinclude the specific infrared command-sequences needed to communicateeach particular control-action to the specific model ofelectronic-device being controlled. The configuration-data may alsoinclude a full set of selection-images or menu-images that may be usedto control each infrared controlled electronic-device in the user'ssystem.

The same or similar procedure as that described in FIG. 3 may also beused for older infrared controllable devices; except that in block 306,the control action(s) are communicated via infrared signaling/codes tothe device-being-controlled.

For example, when the user is controlling an older infrared controllabletelevision: the processing 106 may place the appropriate selection-menusfor the television on one or more of the active-displays 101 using theconfiguration-data. The user may then make selections on one or moreselection-menus to control the television (e.g., change channel orvolume). The processing 106 then determines the desired usercontrol-action(s). The processing may then use the configuration-data togenerate the appropriate infrared command-sequence for the user'sspecific model television in-order to cause the desiredcontrol-action(s).

In another alternative embodiment, the remote-controller 105 may have aninfrared output aperture, in addition to the pointer aperture 102. Thecircuitry for the infrared output may be similar to existing infraredremote control devices (e.g., a television or DVD remote-controller).When an older device needs to be controlled, the processing 106 mayutilize the infrared output [in the remote-controller 105], in-order tosend the infrared control-codes that are compatible with the olderinfrared-controlled electronic device to cause the desiredcontrol-action. The infrared output aperture may be aligned to outputthe infrared-beam in the same direct as the pointer 102 output. In someembodiments, the output infrared beam-width may be wider than existinginfrared-controllers, so that the infrared-beam may still be received bydevices that are positioned near/around the detection-region 921[display region], while the pointer-spot 112 is designating objectswithin the detection-region 921.

One embodiment may be described by the following:

A method for remote control or interactive control, comprising:

-   -   providing a display region;    -   providing a pointer that can place a spot within said display        region, wherein at least a portion of said pointer spot is        visible to a user;    -   providing detectors that are connected or attached to said        display region, wherein said detectors can sense the location of        said pointer spot within said display region;    -   analyzing the response from said detectors and controlling or        affecting at least one electronic-device;    -   whereby a user may control or interact-with one or more        electronic devices by designating with said pointer spot within        said display region.

The above embodiment may be further enhanced by the addition of any oneof the following elements or features:

-   -   wherein said display region contains one or more objects that        may be selected or interacted with. Or    -   wherein said display region contains one or more active-displays        or fixed-displays or real world objects. Or    -   wherein at least some of said detectors are located in one or        more cameras that are connected or attached to said display        region. Or    -   wherein at least some of said detectors are located in one or        more cameras that are mounted on a structure, wherein at least        one dimension of said structure is adjustable to accommodate        different sizes of said display region, wherein said structure        is attached to said display region. Or    -   wherein at least some of said detectors are distributed across        at least a portion of said display region. Or    -   wherein at least some of said detectors are mounted on a        structure; wherein said structure is connected or attached to        said display region. Or    -   further comprising providing a user interface for conveying a        user request or user action to said analyzing or controlling. Or    -   wherein said sensing of the location of said pointer-spot by        said detectors is initiated by user action or activity. Or    -   wherein user action or user activity causes one or more objects        to be displayed within said display region. Or    -   wherein said analyzing includes determining the location of said        pointer spot within said display region. Or    -   wherein said analyzing includes determining one or more objects        that said pointer spot was designating. Or    -   wherein said controlling includes communicating commands to at        least one electronic-device;    -   wherein said commands are at least partly based upon a location        of the pointer-spot within said display region. Or    -   wherein said controlling includes altering the content or        objects within said display region, at least partially based        upon the object or objects that the user designated with said        pointer spot. Or    -   wherein said pointer is a light emitting source or a laser        pointer or a focused light beam or a spot of light or a spot of        shadow.

Another embodiment may be described by the following:

An apparatus, comprising:

-   -   a display region;    -   a pointer capable of designating a spot in said display region,        wherein at least a portion of said pointer spot is visible to a        user;    -   detectors that are connected or attached to said display region,        wherein said detectors can sense the location of said pointer        spot within said display region;    -   a processing-function to analyze the response from said        detectors, and to control or affect at least one        electronic-device;    -   whereby a user may control or interact-with; one or more        electronic devices, by placing the pointer spot on or near        objects within said display region.

The above embodiment may be further enhanced by the addition of any oneof the following elements or features:

-   -   further comprising a user interface for conveying a user request        or user action to said processing. Or    -   wherein said display region includes one or more        active-displays; fixed-displays; or real world objects. Or    -   wherein one or more objects appear in said display region when        initiated by user action or activity. Or    -   wherein said sensing by said detectors is initiated by user        action or activity. Or    -   wherein said control includes changing objects in said display        region, based upon the object or objects that the user        designated with said pointer spot.

One embodiment may be described by the following:

A display configuration, comprising:

-   -   a display region;    -   detectors connected or attached to said display region; wherein        said detectors can sense the location of a pointer spot placed        at locations within said display region, wherein at least a        portion of said spot is visible to a user;    -   a processing-function to analyze the response from said        detectors, and to control or affect at least one        electronic-device;    -   whereby a user may control or interact-with; one or more        electronic devices, by placing the pointer spot on or near        objects within said display region.

The above embodiment may be further enhanced by the addition of any oneof the following elements or features:

-   -   wherein at least some of said detectors are located in one or        more cameras that are connected to said display region. Or    -   wherein at least some of said detectors are distributed across        at least a portion of said display region. Or    -   wherein at least some of said detectors are mounted on a        structure; wherein said structure is connected or attached to        said display region. Or    -   wherein said display region contains one or more active-displays        or fixed-displays or real world objects.

Not Limited to Detailed Illustrations:

It should be noted that the above mentioned embodiments illustraterather than limit the invention, and that those skilled in the art willbe able to design many alternative embodiments without departing fromthe scope of the invention as defined in the claims.

To satisfy the requirements for enablement, this disclosure may containone or more embodiments which illustrate a particular detailedimplementation and use. For clarity simplified examples may be chosenfor illustration. A detailed illustration often requires choosing onlyone of a plurality of equivalent detail approaches to describe. Whenterms such as “shall”, “should”, “is”, “are” appear in this description,they should only be interpreted as limitations/requirements for thepurpose of maintaining compatibility/consistency between theelements/parameters of the particular detailed illustration beingdescribed. Such terms should not be interpreted as limitations orrequirements on the scope of the general inventive concept as disclosedin its entirety.

For example, if element “A”, in a detailed embodiment, is shown ashaving a certain detailed configuration, then mating element “B” in thatdetailed example may need to have corresponding “mating” limitations;in-order to be compatible and/or interoperable with the detailed element“A”. Such mating-limitations on element “B” for compatibility within adetailed illustration, do not define limitations on element “B” withinall the possible embodiments that fall within the scope of theinvention. If an alternate configuration of element “A” had beenselected for illustration purposes, the detail implementationrequirements on element “B” for compatibility and interoperable with theelement “A” may be very different.

In general, the detailed implementations for the elements of theinvention may have many alternate implementations that accomplish thesame functional result/objective and are within the scope of the generalinventive concept.

1. Apparatus for remotely controlling one or more electronic devices,comprising: a display-region; wherein detectors are physicallyattached-to or physically integrated-into said display-region; whereinsaid detectors image the same locations in said display-region, whensaid display-region is moved or re-oriented; a ring-shaped unit havingan opening sized to be worn on an index finger of a user; alight-emitting pointer incorporated as part of said ring-shaped unit,wherein a pointer-spot of visible light is projected when said pointeris activated; a wireless interface incorporated as part of saidring-shaped unit; a single actuator incorporated as part of saidring-shaped unit, so that when the user is wearing said ring-shaped uniton their index finger, the user is able to actuate the actuator with athumb of a hand the ring-shaped unit is being worn on; wherein saidactuator only moves in a substantially single axis when actuated by auser; wherein, when a first action on said actuator by the user isdetected, the light-emitting pointer is activated and a control signalis sent across the wireless interface to turn-on an activecontrol-display in said display-region, and to cause selectable-objectsto appear in said display-region; wherein, when the user is wearing saidring-shaped unit on their index finger, the user is able to place thepointer-spot onto a selectable-object in said display-region while theuser is able to activate the actuator with a second action; wherein whena second action on said actuator by the user is detected, a controlsignal is sent across the wireless interface to cause said detectors tocapture an image or images of said pointer-spot in said display-region,and to cause a processor to analyze said captured image or images todetermine which selectable-object, the pointer-spot is on or near;wherein when a selectable-object is determined to have been selected bythe pointer-spot, the processor sends control information to affect atleast one electronic device.
 2. Apparatus as in claim 1 wherein the sizeof said opening is adjustable to accommodate the finger of a pluralityof different users that have different sizes.
 3. Apparatus as in claim 1wherein said first action is to push on the actuator and wherein saidsecond action is to stop pushing on said actuator.
 4. Apparatus as inclaim 1 wherein said first action is to push the actuator from an offposition into a first position and the said second action is to push theactuator further into a second position.
 5. Apparatus as in claim 1wherein the actuator only moves along one physical axis in response tothe user pressing or releasing the actuator.
 6. Apparatus as in claim 1wherein the ring-shaped unit has only a single actuator which is a pushand release button.
 7. Apparatus as in claim 1 wherein the ring-shapedunit has only one push button; wherein the push button has an offposition and a select position; wherein the push button automaticallyreturns to the off position when not being pressed on by a user. 8.Apparatus as in claim 1 wherein said detectors are one or more camerasthat are physically attached-to or physically integrated-into saiddisplay-region; wherein said detectors image the same locations in saiddisplay-region when said display-region is moved or re-oriented. 9.Apparatus as in claim 1 wherein at least some of said detectors aremounted-on or incorporated-into a structure; wherein said structure isphysically attached to said display-region; wherein said detectors imagethe same locations in said display-region when said display-region ismoved or re-oriented.
 10. Apparatus as in claim 1 wherein at least someof said detectors are located in one or more cameras that are mounted ona structure, wherein at least one dimension of said structure isadjustable to accommodate different sizes of said display-region,wherein the dimension of said structure is adjusted as said structure isattached to said display-region.
 11. Apparatus as in claim 1 whereinresponsive to the selection of a selectable-object displayed on anactive-display in said display-region, some of the selectable-objectsshown on the active-display are removed and other selectable-objects aredisplayed on the active-display.
 12. Apparatus as in claim 1 wherein anactive-display in said display-region displays a plurality ofselectable-objects that represent control buttons for a plurality ofelectronic devices; wherein said control information is sent across anetwork to control said plurality of the electronic devices. 13.Apparatus as in claim 1 wherein some of the selectable-objects in saiddisplay-region represent control buttons for a plurality of electronicentertainment devices; and said control information is sent across anetwork to control a plurality of the electronic entertainment devices.14. Apparatus as in claim 1 further comprising: a network capable offorwarding said control information from said processor to a pluralityof user entertainment devices.
 15. Apparatus as in claim 1 furtherincluding a network interconnecting a plurality of electronic-devices,wherein said control information is sent to a plurality of electronicdevices that are attached to the network.
 16. Apparatus as in claim 1further including a network interconnecting a plurality ofelectronic-devices; wherein said control information is sent to aplurality of electronic devices that are attached to the network;wherein the network automatically incorporates a newly connectedelectronic-device into the network, or automatically removes a newlydisconnected electronic-device from the network.
 17. Apparatus as inclaim 1 further including a network interconnecting a plurality ofelectronic-devices; wherein said control information is sent to aplurality of electronic devices that are attached to the network;wherein the network automatically incorporates a newly connectedelectronic-device into the network, or automatically removes a newlydisconnected electronic-device from the network; whereinselectable-objects for a newly connected electronic-device areautomatically included in the display-region; wherein selectable-objectsfor a newly disconnected electronic-device are automatically removedfrom the display-region.
 18. Apparatus as in claim 1 wherein theselectable-object that the pointer-spot is on or near, is determinedwithout displaying or moving a cursor on an active-display. 19.Apparatus as in claim 1 wherein said analysis by said processor, usesinformation that defines spatial locations of the selectable-objects, insaid display-region.
 20. Apparatus as in claim 1 wherein said analysisby said processor, utilizes information that defines areas or locationsin said display-region that are seen by individual detectors. 21.Apparatus as in claim 1 wherein if prior to said first action, anactive-display in said display-region, is unpowered or is not displayingselectable-objects, then said first action causes the active-display insaid display-region to power-on and to display selectable-objects. 22.Apparatus as in claim 1 wherein if prior to said first action,entertainment content without selectable-objects is being displayed onan active-display in said display-region; then said first action causesthe active-display to display selectable-objects.
 23. Apparatus as inclaim 1 wherein if prior to said first action, entertainment contentwithout a selectable-object is displayed on an active-display in saiddisplay-region; and in response to detecting said first action, aplurality of selectable-objects are displayed on said active-display;and if said second action is not detected within a defined time, thenthe selectable-objects are automatically removed from saidactive-display and said content without selectable-objects is displayedon said active-display.
 24. Apparatus as in claim 1: wherein thelocation in said display that each detector images is set duringmanufacture of said display; wherein said detectors image the samelocations in said display, when said display is moved or re-oriented.25. Apparatus as in claim 1: wherein said detectors are limited todetecting wavelengths that correspond to said light-emitting pointer.26. Apparatus as in claim 1: wherein fixed-objects, representingcontrols for an electronic-device, are in said display-region.
 27. Amethod for remotely controlling one or more electronic devices,comprising: providing a display-region; wherein detectors are physicallyattached-to or physically integrated-into said display-region; whereinsaid detectors image the same locations in said display-region, whensaid display-region is moved or re-oriented; providing a ring-shapedunit having an opening sized to be worn on an index finger of a user;wherein said ring-shaped unit includes: a light-emitting pointerincorporated as part of said ring-shaped unit, wherein a pointer-spot ofvisible light is projected when said pointer is activated; a wirelessinterface incorporated as part of said ring-shaped unit; a singleactuator incorporated as part of said ring-shaped unit, wherein when theuser is wearing said ring-shaped unit on their index finger, the user isable to place the pointer-spot onto a selectable-object in saiddisplay-region while the user is able to activate the actuator; whereinsaid actuator only moves in a substantially single axis when actuated bya user; when a first action on said actuator by the user is detected,activating the light-emitting pointer and sending a control signalacross the wireless interface to turn-on an active control-display insaid display-region, and to cause selectable-objects to appear in saiddisplay-region; when a second action on said actuator by the user isdetected, sending a control signal across the wireless interface tocause detectors to capture an image or images of said pointer-spot insaid display-region; analyzing said captured image or images todetermine which selectable-object, the pointer-spot is on or near; andwhen said analyzing determines that a selectable-object is selected bythe pointer-spot, sending control information to affect at least oneelectronic device.
 28. A method as in claim 27 wherein theselectable-object that the pointer-spot is on or near, is determinedwithout displaying or moving a cursor on an active-display.
 29. A methodas in claim 27 further comprising: sending said control informationacross a network to affect a plurality of electronic devices that areattached to the network.
 30. Apparatus for remotely controlling aplurality of electronic devices, comprising: a display-region withdetectors that are physically attached-to or integrated-into saiddisplay-region; wherein when the display-region is moved or re-oriented,said detectors image the same locations in said display-region; aring-shaped unit having an opening sized to be worn on an index fingerof a user; a light-emitting pointer incorporated as part of saidring-shaped unit, wherein a pointer-spot of visible light is projectedwhen said pointer is activated; wherein at least a portion of saidpointer-spot will be visible to the user when reflecting or scatteringoff said display-region; wherein said detectors are able to image a spotof light from said light-emitting pointer that is shining in saiddisplay-region; a single actuator incorporated as part of saidring-shaped unit, wherein when the user is wearing said ring-shaped uniton their index finger, the user is able to place the pointer-spot onto aselectable-object in said display-region while the user is able toactivate the actuator; wherein said actuator only moves in asubstantially single axis when actuated by a user; a wireless interfaceincorporated as part of said ring-shaped unit; wherein when a firstaction by the user on said actuator is detected, the light-emittingpointer is activated and a control signal is sent across the wirelessinterface to turn-on an active control-display in said display-region,and to cause selectable-objects to appear in said display-region;wherein when additional actions by the user on said actuator isdetected, a control signal is sent across the wireless interface tocause said detectors to capture an image or images of said pointer-spotin said display-region; a processor to analyze said captured image orimages to determine which selectable-object, the pointer-spot is on ornear; wherein dependent on which selectable-object or selectable-objectsthe pointer spot is on or near, the processor sends control informationacross a network to affect a plurality of electronic devices. 31.Apparatus as in claim 30 wherein the ring-shaped unit has only a singleactuator that only moves in a substantially single axis when actuated bya user.
 32. Apparatus as in claim 30 wherein the selectable-object thatthe pointer-spot is on or near, is determined without displaying ormoving a cursor on an active-display.